Addressing Partitioned Arrays in Distributed Memory Multiprocessors – the Software Virtual Memory Approach

Partitioning distributed arrays to ensure locality of reference is widely recognized as being critical in obtaining good performance on distributed memory multiprocessors. Data partitioning is the process of tiling data arrays and placing the tiles in memory such that a maximum number of data accesses are satisfied from local memory. Unfortunately, data partitioning makes it difficult to physically locate an element of a distributed array. Data tiles with complicated shapes, such as hyperparallelepipeds, exacerbate this addressing problem. In this paper we propose a simple scheme called software virtual memory that allows flexible addressing of partitioned arrays with low runtime overhead. Software virtual memory implements address translation in software using small, one-dimensional pages, and a compiler-generated software page map. Because page sizes are chosen by the compiler, arbitrarily complex data tiles can be used to maximize locality, and because the pages are one-dimensional, runtime address computations are simple and efficient. One-dimensional pages also ensure that software virtual memory is more efficient than simple blocking for rectangular data tiles. Software virtual memory provides good locality for complicated compile-time partitions, thus enabling the use of sophisticated partitioning schemes appearing in recent literature. Software virtual memory can also be used in systems that provide hardware support for virtual memory. Although hardware virtual memory, when used exclusively, eliminates runtime overhead for addressing, we demonstrate that it does not preserve locality of reference to the same extent as software virtual memory.